STATISTICAL PROPERTIES OF LEAST SQUARES ESTIMATORS. x, where. = y - ˆ " 1

Transcription

1 STATISTICAL PROPERTIES OF LEAST SQUARES ESTIMATORS Recall Assumpto E(Y x) η 0 + η x (lear codtoal mea fucto) Data (x, y ), (x 2, y 2 ),, (x, y ) Least squares estmator ˆ E (Y x) ˆ " 0 + ˆ " x, where ˆ " SXY ˆ " 0 y - ˆ " x ( x - x ) 2 x ( x - x ) SXY ( x - x ) (y - y ) ( x - x ) y Commets So far we have t used ay assumptos about codtoal varace 2 If our data were the etre populato, we could also use the same least squares procedure to ft a approxmate le to the codtoal sample meas 3 Or, f we just had data, we could ft a le to the data, but othg could be ferred beyod the data 4 (Assumg aga that we have a smple radom sample from the populato) If we also assume e x (equvaletly, Y x) s ormal wth costat varace, the the least squares estmates are the same as the maxmum lkelhood estmates of η 0 ad η Propertes of ˆ " 0 ad ˆ " ) ˆ " SXY (x " x )y where c (x " x ) (x " x ) y " c y Thus If the x 's are fxed (as the blood lactc acd example), the ˆ " s a lear combato of the y 's Note Here we wat to thk of each y as a radom varable wth dstrbuto Y x Thus, f the y s are depedet ad each Y x s ormal, the ˆ " s also ormal If the Y x 's are ot ormal but s large, the ˆ " s approxmately ormal Ths wll allow us to do ferece o ˆ " (Detals later) 2) c (x " x ) (x " x ) 0 (as see establshg the alterate expresso for )

2 2 3) x c x (x " x ) x (x " x ) Remark Recall the somewhat aalogous propertes for the resduals 4) ˆ " 0 y - ˆ " x hece " y - c y ˆ e " x ( " c x )y, also a lear combato of the y 's, 5) The sum of the coeffcets (4) s ( " c x ) ( ) " x c ( ) " x 0 Samplg dstrbutos of ˆ " 0 ad ˆ " Cosder x,, x as fxed (e, codto o x,, x ) Model Assumptos ("The" Smple Lear Regresso Model Verso 3) E(Y x) η 0 + η x (lear codtoal mea fucto) Var(Y x) σ 2 (Equvaletly, Var(e x) σ 2 ) (costat varace) (NEW) y,, y are depedet observatos (depedece) The ew assumpto meas we ca cosder y,, y as comg from depedet radom varables Y,, Y, where Y has the dstrbuto of Y x Commet We do ot assume that the x 's are dstct If, for example, x x 2, the we are assumg that y ad y 2 are depedet observatos from the same codtoal dstrbuto Y x Sce Y,, Y are radom varables, so s ˆ " -- but t depeds o the choce of x,, x, so we ca talk about the codtoal dstrbuto ˆ " x,, x Expected value of ˆ " (as the y 's vary) E( ˆ " x,, x ) E( " c Y x,, x ) c E(Y x,, x ) c E(Y x ) (sce Y depeds oly o x ) c (η 0 + η x ) (model assumpto) η 0 c + η c x η η η Thus ˆ " s a ubased estmator of η

3 3 Varace of ˆ " (as the y 's vary) Var( ˆ " x,, x ) Var( " c Y x,, x ) c 2 Var(Y x,, x ) c 2 Var(Y x ) (sce y depeds oly o x ) c 2 σ 2 σ 2 2 c σ 2 (x " x ) 2 & % ( (defto of c $ ' ) " 2 $ (x ( ) 2 x ) 2 " 2 For short Var( ˆ " ) " 2 sd( ˆ " ) " Commets Ths s vaguely aalogous to the samplg stadard devato for a mea y populato stadard devato sd (estmator) somethg However, here the "somethg," amely, s more complcated But we ca stll aalyze ths formula to see how the stadard devato vares wth the codtos of samplg For y, the deomator s the square root of, so we see that as becomes larger, the samplg stadard devato of y gets smaller Here, recallg that ( x - x ) 2, we reaso that If the x 's are far from x (e, spread out), s, so sd( ˆ " ) s If the x 's are close to x (e, close together), s, so sd( ˆ " ) s Thus f you are desgg a expermet, choosg the x 's to be from ther mea wll result a more precse estmate of ˆ " (Assumg all the model codtos ft) Expected value ad varace of ˆ " 0 Usg the formula ˆ " 0 ( " c x )y, calculatos (left to the terested studet) smlar to those for ˆ " wll show E( ˆ " 0 ) η 0 (So ˆ " 0 s a ubased estmator of η 0 )

4 4 Var ( ˆ " 0 ) " 2 + x 2 & % $ ( ', so sd ( ˆ " 0 ) " + x 2 Aalyzg the varace formula A larger x gves a varace for ˆ " 0 Does ths agree wth tuto? A larger sample sze teds to gve a varace for ˆ " 0 The varace of ˆ " 0 s (except whe x < ) tha the varace of ˆ " Does ths agree wth tuto? The spread of the x 's affects the varace of ˆ " 0 the same way t affects the varace of ˆ " Covarace of ˆ " 0 ad ˆ " Smlar calculatos (left to the terested studet) wll show x Cov( ˆ " 0, ˆ " ) " 2 Thus ˆ " 0 ad ˆ " are ot depedet (except possbly whe ) Does ths agree wth tuto? The sg of Cov( ˆ " 0, ˆ " ) s opposte that of x Does ths agree wth tuto? Estmatg σ 2 To use the varace formulas above for ferece, we eed to estmate σ 2 ( Var(Y x ), the same for all ) Frst, some plausble reasog If we had lots of observatos y, y 2,,y m from Y x, the we could use the uvarate stadard devato m (y m " j " y ) 2 j of these m observatos to estmate σ 2 (Here y s the mea of y, y 2,,y m, whch would be our best estmate of E(Y x ) just usg y, y 2,,y m ) We do't typcally have lots of y's from oe x, so we mght try (reasog that E ˆ (Y x ) ) s our best estmate of E(Y x )) [y " " E ˆ (Y x )] 2

5 5 " 2 e ˆ " RSS However (just as the uvarate case, we eed a deomator - to get a ubased estmator), a legthy calculato (omtted) wll show that E(RSS x,, x ) (-2) σ 2 (where the expected value s over all samples of the y 's wth the x 's fxed) Thus we use the estmate ˆ " 2 " 2 RSS to get a ubased estmator for σ 2 E( ˆ " 2 x,, x ) σ 2 [If you lke to thk heurstcally terms of losg oe degree of freedom for each calculato from data volved the estmator, ths makes sese Both ˆ " 0 ad ˆ " eed to be calculated from the data to get RSS] Stadard Errors for ˆ " 0 ad ˆ " Usg ˆ " RSS " 2 as a estmate of σ the formulas for sd ( ˆ " 0 ) ad sd( ˆ " ), we obta the stadard errors ad se ( ˆ " ) se( ˆ " 0 ) ˆ " ˆ " + x 2 as estmates of sd ( ˆ " ) ad sd ( ˆ " 0 ), respectvely